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§1. Introduction and Results. Let A g be the moduli space of principally polarized 
Abelian varieties and E be the Hodge bundle. 

1.1 Main Theorem. Let X be a compact subvariety of A g eg) C such that q(E)|x = 
in H 2t (X, R) for some g > i > 0. Then 

(1) dim(X) < 

with strict inequality ifi>3. 

The Grothendieck Riemann Roch theorem implies that c g (E) is trivial on A g , see 
[G99,2.2], thus we have the following corollary, conjectured by Oort, [OG,3.5]: 

1.2 Corollary. There is no compact codimension g subvariety of A g <g> C, for g > 3. 

Corollary 1.2 is striking in that it fails in positive characteristic: Oort has shown 
that the locus Z C A g <g> E p of Abelian varieties of p-rank zero is complete and pure 
codimension g. 

We also obtain: 

1.2.1 Corollary. There is no compact codimension g subvariety of M° <g> C for g > 3. 

This is a formal consequence of (the g = 3 case of) (1.2), and the bounds of [Diaz87] 
on dimensions of compact subvarieties of M g , see §8. 

Inequality (1) has been obtained previously, in all characteristics, by van der Geer, 
[G99]. The improvement to strict inequality may seem rather humble, but in relation 
to Faber's conjectures, it is quite significant: 

Faber has made the surprising conjecture that the subring 1Z*(M g ) C CH*(M 9 )q ® 
Q generated by tautological classes looks like the rational cohomology of a smooth 
projective variety of dimension g—2, e.g., it satisfies Poincare duality and Grothendieck's 
standard conjectures. See [Faber99] for the precise statement, and very compelling 
evidence. One naturally wonders if there is a projective g — 2 dimensional variety with 
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the right cohomology. The form of the conjectures, specifically his evaluation maps, 
[FP00,0.4], strongly suggest looking for a compact codimension 2g — 1 subvariety of 
M g , whose cohomology class in M g is a multiple of c 5 (E)c 3 _i(E). There are analogous 
speculations for M g (stable curves of compact type), [FP00,0.5], that lead one to hope 
for a compact subvariety of codim g, with cohomology class a multiple of c g (E). Since E 
comes from A g , it is natural to hope that the subvarieties do as well. More compellingly, 
there are natural analogs of Faber's conjectures for A g , and A° (the open subset of 
A g whose complement correspond to products of lower dimensional polarized varieties 
- note the image of the Torelli map is is a closed subset of A°), and corresponding 
evaluation maps which themselves raise the hope of compact subvarieties, of codimension 
g in A g , and of codimension 2g — 1 in A°. Indeed the tautological ring lZ*(A g ) has 
been computed, [G99] and [EV02], and the analog of Faber's conjecture holds, so if in 
particular one has a compact subvariety Z C A g with the desired cohomology class, 
then Poincare duality already gives that lZ*(A g ) — >■ H*(Z,Q) is injective. 

Unfortunately, by Corollary 1.2 the hoped for subvariety of A g does not exist in 
characteristic zero. Theorem 1.1 suggests a similar result may hold for A°: It is natural 
to wonder if c g _i(E) vanishes on A°. If it does, then by Theorem 1.1, with i = g — 1, 
there is no compact subvariety of A° <g> C of codimension 2g — 1, for g > 4. 

As noted, Oort's example Z C A g ®W p violates the analog of (1.2) for characteristic p. 
As Oort pointed out to us, since it is known that Z does not lift to char 0, uniqueness of 
Z would give a different, purely char p proof of (1.2). In this sense (1.2) is evidence for 
this uniqueness, which would obviously be attractive from the point of view of Faber's 
conjecture. In §7 we point out an intriguing parallel between our proof in char and 
Oort's example; perhaps it can be exploited. 

Finally, Theorem 1.1 is sharp for i = 2 and i = 3. There exists a compact surface 
Z C A 3 and a compact curve C C A 2 . Taking products with fixed elliptic curves one 
then obtains a compact surface Z C A g , g > 3 and a compact curve C C A g , g > 2 
with c 3 (E)| z = 0, c 2 (E)| c = 0. 

After submitting this paper for publication we learned from E. Colombo that the 
i = g case of Corollary (2.6), the crucial technical step in the proof of Corollary (1.2), 
has been previously obtained by E. Izadi, [Izadi98], using methods of Colombo and 
Pirola, [CP90]. The main argument of Colombo-Pirola and Izadi is entirely different 
from ours (and quite ingenious). It is more elementary (no use is made of the vanishing 
of c g (E)) and also more algebaic, so might be useful for showing uniqueness of Oort's 
Z c A g ® F p . 
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§2. Proof of Main Theorem. 

In this section we give the main line of our argument for Theorem 1.1. The various 
constituent results will be proved in subsequent sections. 

We use throughout the (orbifold) cover q : H g — > A g by Siegel's generalized upper 
half space. (We could as well replace A g by a finite branched cover, and q by an honest 
(unbranched) cover. Above and throughout the paper any statement about A g should 
be interpreted in the orbifold sense.) 7i g is by definition the space of symmetric complex 
g x g matrices with positive definite imaginary part, while S g denotes the space of all 
symmetric g x g complex matrices. 

2.1. C hern Forms. E has a tautological Hermitian metric: A point of A g is given by a 
pair (V/L, <, >), where L C V is a lattice in a g-dim complex vector space, and <, > is 
a positive definite Hermitian form (such that im <, > \l is integral and unimodular). E 
is the Hermitian bundle with fibre V* and metric dual to <, >. (We abuse notation and 
denote this dual metric, and any other metric induced from <, >, by <, >). Griffiths 
showed that the curvature of (E, <, >) is non-negative, [Griffiths84] . Our proof is based 
on the tension between this non-negativity and the vanishing of c 5 (E) noted above: As 
E > it's natural to hope that Cfc(E) is represented by a non- negative form c^. (Recall 
a real (k, k) form is called non-negative if its restriction to every complex /c-plane in 
the tangent bundle is a non- negative multiple of the volume form.) As we indicate in a 
moment, this is indeed the case. Thus if Y C A g is a compact /c-fold and 



then Ck\y vanishes identically (on the smooth locus), and we get pointwise information 
on the tangent space of Y. 

Here is how we obtain By Hodge theory, E sits in an exact sequence 



(2) 




(3) 



0^E^H^E*^0, 
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where HI is a flat bundle (with fibre H 1 (A, C) at the point [A] e A g given by the Abelian 
variety A) , whence the equality of cohomology classes 



(4) c(E) • c(E*) = 1. 

Thus c(E) is represented by the inverse Chern form of (E*, <, >). This is computed as 
follows: 

2.2 Segre forms. Let (F, <, >) be a rank g Hermitian bundle, on a complex manifold 
£?, with curvature fi, let G = ^qfi, and let p : F(F) — > B be the projective bundle of 
lines in F. Let 

(5) A : T P ( F ) — > T P ( F )/ B 

(where Tp(F)/B is the vertical tangent space, i.e., the kernel of dp) be the second fun- 
damental form for the tautological sublinebundle O(-l) C p*(F). Let s(F, <, >) = 
Sfc(F, <, >) denote the Segre from, i.e., 

(6) s k (F)= P 4 Cl (0(l),<,>y- k+1 ). 



2.3 Theorem. The kernel of A defines a horizontal subspace, complementary to the 
vertical tangent space, which is identified with p*(Ts) by dp. Under this splitting of 
Tp(F) the first Chern form of 0{1) at a line L C Ft,, spanned by a unit vector v, is 

(7) FS+<Gv,v>, 

where FS is the Fubini-Study form ( on the vertical tangent space, normalized to repre- 
sent the hyperplace class). The Segre form Sk(F, <, >) equals 

and the total Segre form s(F, <, >) equals c(F, <, >) _1 (pointwise!) in the algebra of 
even degree forms. 

It is well known that the Segre class represents the inverse Chern class in cohomology, 
see e.g. [BT82,pg 269], [Fulton80,3.1]. We do not know if this equality at the level of 
forms, or the rather canonical expressions above, have been previously observed. 

The tangent space T p A g is canonically identified with 5 , (E p ,E*), the space of sym- 
metric linear maps from E p to its dual. Theorem 2.3 applied to (E*, <, >), together 

with Griffiths' formula for the curvature, imply: 
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2.4 Corollary. The k th Chern class of E is represented by dk '■= s/-(E*,<,>). The 
form Cfc is non-negative and vanishes on a (complex) i-plane Y in T p A g = 5(E p , E*) if 
and only if, for every v G E p , £/ie evaluation map e v : Y" — > E* /az/s to 6e injective. 

For each line L C E p , choose O^wGi. Let <, >l on >S(E P ,E*) 6e £/ie Hermitian 
form e* w {<, > E .)/ < «;,«; >, i.e., 

/ Q x , i ^ < e w (a),e w (fr) > 

(9) < a, 6 > L = . 

< w, w > 

Up to a positive constant, the restriction ofdk to p is the average of the k th wedge powers 
of —Im(<, >l) over all lines L C E p . 

We prove (2.3-4) in §3. 

Given compact X C A g with Cj(E)|x = 0, we may apply Corollary 2.4 to every 
i-plane in the tangent space at smooth points of X. (Since A g is quasi-projective, each 
i-plane is realized as a tangent plane to a compact i-dimensional subvariety.) 

We reformulate this using the following linear algebra result, proved in §4. 

2.5 Theorem. Let V be a complex vector space of dimension g > 1. Let i be an integer 
g > i > 0. Let X C S(V,V*) be a linear subspace of dimension at least max(i, ^ ) 
such that for each v G V and every i- dimensional subspace Y C X the evaluation map 
e v : Y — > V* fails to be injective. 

Then i > 3, X = W 1 - for some g — i + 1-dimensional linear subspace W C V , and 
X has dimension exactly ■ 

Here W 1 - C S(V, V*) is the set of maps whose kernels contain W. 
Combining this with Corollary 2.4 we find: 

2.6 Corollary. Let g > i > 3. Let X C A g be a compact subvariety of dimension at 
least — 2 1 ' ) such that q(E)|x = 0. Then X has dimension exactly t ^ 2 — . Furthermore, 
at each smooth point p G X , T P X = W 1 - C 5 , (E P ,E*) for some g — i + 1 dimensional 
subspace W(p) C E p . 

Let X C A g be as in Corollary 2.6, and let X be an irreducible component of q~ 1 (X). 
By Corollary 2.6, 

(10) T A X = W(A) 1 - c S g 



for each smooth A G X. 
In §5 we prove: 
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2.7 Theorem. The subspace W(A) is constant (and henceforth denoted W). If X is 
an irreducible component of q~ 1 (X), then for some fixed r G H g , X C H g is the affine 
space 

(11) X(W, r) := {M G H g \M\ w = r\ w }. 



Finally, in §6 we show: 
2.8 Theorem. The image of X(W,r) in A g cannot be compact. 
§3 Segre Forms. 

Proof of Theorem 2. 3. The restriction of A to the vertical tangent space is the identity, so 
A is indeed surjective. Now equation (7) follows by the definition of A, see [GH78,pg 78]. 
What remains is to show that the pointwise inverse of the total Chern form c(F, <, >) 
is given by the formula (8) , and equals the Segre form. 

If M is a positive Hermitian form on a g-dimensional complex vector space V with 
inner product <, >, then 



(12) 7r- 5 det(M) / e~ <Mv ' v> dVol 



v 



This identity remains true if M takes values in the commutative ring of even degree 
forms at a point, as long as the scalar part of M remains positive-definite. 



Now let V = F p , and let M = I - G. Then 

[c(F, <,>)]-* =tt~ 9 I e- <(I - G ^ v ' v> dVol Fp 

J F„ 



-TV 9 



E < > e-^dVoh 



k 



P k 



k\ 



v /g + fc-l\ 1 f ( <Gv,v> \ k 
^\ k JVol(F(F p ))J ¥{Fp) \ <v,v> I ' ° 



k 



< V, V > 



In going from the second line to the third, we integrate over radius, using the facts that 
/ °° r 2k e -r 2 r 2 9 -i dr = ^ + g _ and that the volume of S 29 ' 1 is 2^/{g - 1)!. 



□ 



Proof of Corollary 2.4- The second paragraph implies the first, since <,>l is semi- 
positive, and its restriction to an i-plane Y fails to be positive iff e„*|y fails to be 
injective. Thus Ck\y is non-negative and vanishes iff e v * fails to be injective for all v. 

What remains, then, is to compute the curvature of E*, from which we obtain the 
Segre form Sk by equation (8). This is given by Griffiths' formula [Griffiths84] , but can 
also be computed from a direct calculation, which we include for the reader's conve- 
nience: The metric on E* is given by the inner product h = (77) _1 , where 77 denotes 
the imaginary part of r, see [Kempf91,pg 59]. Thus the curvature is given by: 

o =d{hr x dh) 

= -d[(dr I )rr 1 } 
(M) =-(dr I )Tr 1 (dF I )rr 1 

= -\{dr)TY\d^)rY\ 

If v e E*, then w := rj x v =< •, v >G E, and 

< Gv,v >=—v t TY 1 {dr)Tj 1 (dT)TY 1 v 

OTT 

(15) =^<e w (dr),e w (dr)> 

Note that < v,v >=< w,w >, and the unit sphere in E* is naturally identified with the 
unit sphere in E p . As a result, averaging < Gv, v > k / < v,v > k over all values of v is 
equivalent to averaging (—Im(<, >L)/47r) fc over all lines in E p . □ 

Remark: It is natural to wonder whether or not the Chern form Cfc(E, <, >) and the 
Segre form Sfc(E*, <, >) are actually equal. We checked that this holds for k = 1, 2, and 
suspect that it holds in general. 

§4 Linear algebra. 

As before, let S(V, V*) be the space of symmetric maps from the vector space V to 

its dual. For each v E V, let v 1 - C S(W, W*) be the set of maps that vanish on v. Let 

Rk C S(W, W*) be the locus of maps whose image has dimension at most k. (This is a 

closed subset, not a vector subspace.) 

We prove Theorem 2.5 in stages, beginning with the following standard fact: 
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4.1 Lemma. Let M G Rk\ Rk-i- Rk is smooth at M with tangent space 



(16) T M Rk = {N E S(V, V*)\N(ker(M)) C im(M)}. 

4.2 Corollary. Let Y C <S(V, y*) 6e a 2-plane spanned by rank 1-maps. Then Y C i?2, 

&«* y £ 

Proo/. Let M, N G i?i be a basis of y, and let TeY. Then ker(M) nker(iV) C ker(T), 
so T G i? 2 - If y C however, then M G TVPfc, so M(ker N) C im(iV). Using that 
they are each rank one, this implies that M and iV are dependent, a contradiction. □ 

4.3 Lemma. Let Y C V*) 6e a linear subspace of R2. Let M &Y, v EV be such 
that M(v)(v) 7^ 0. Then v 1 - fl Y C -Ri and has dimension at most one. 

Proof. Suppose iV G v 1 - fl Y has rank 2. Since M G T N R 2 , Lemma 4.1 implies that 
M(v) = N(w) for some w E V. But then M(v)(v) = N(v)(w) = 0, a contradiction. 
Thus v 1 - n y C Now apply Corollary 4.2 to ^HF. □ 

It will be convenient to prove the cases % = 1, 2 of Theorem 2.5 separately: 

4.4 Proposition. If i <3, then the hypotheses of Theorem 2.5 are never met. 

Proof. We consider an i-dimensional subspace Y on which, for all nonzero v, e v fails to 
be injective. That is, every vector in V is in the kernel of some nonzero element of Y, 
and we have 

(17) V= |J ker(Q). 

QeP(Y) 

If i = 1, this means that all of V is in the kernel of a nonzero matrix, clearly impossible. 
If i = 2, this means that a (7-dimensional vector space is the union of a 1-dimensional 
family of proper subspaces. Thus almost every element of Y has a (g — 1) -dimensional 
kernel, i.e., is of rank 1. But this contradicts Corollary 4.2. □ 

4.5 Proposition. Let i > 3. If Theorem 2.5 holds for all spaces X of dimension 
equal to i(i — l)/2, then the hypotheses of the theorem cannot be satisfied by any X of 
dimension greater than i(i — l)/2. 

Proof. Let X have dimension greater than — l)/2, satisfying the hypotheses of Theo- 
rem 2.5, and let X' be a subspace of dimension — 1)/2. The hypotheses of the theorem 

then apply to X' ', so X' C ly- 1 for some g — i + 1 dimensional subspace W CV. Since 

8 



their dimensions are equal, X' = W^, so we can find % — 1 rank-1 elements xi, . . .Xi-\ 
of X' whose ranges are linearly independent. Let Xi E X — X' ', and let Y be the span 
of xi, . . . , x n . It is then straightforward to find a vector v E V for which e v is injective 
on Y, which is a contradiction. 

In light of Proposition 4.5, we will henceforth only consider spaces X of dimension 
equal to i(i — l)/2. 

4.6 Proposition. Theorem 2.5 holds for i = 3. 

Proof. Since i = i(z — l)/2 = 3, the only i-dimensional subspace F of X is X itself. We 
first show that F C i?2- This follows from equation (17) by counting dimensions. If 
Y E\ R 2 , then there must exist a codimension-one locus Z' C P(Y) of rank 1 quadrics, 
and correspondingly a codim one closed cone Z C Y n But then, by Corollary 4.2, 
y C i?2, which is a contradiction. 

Having shown that F C i?2, we use Lemma 4.3 to construct rank-1 elements of Y. 
Since Y is nonzero, we can find M G Y and v G F such that M (v )(v ) ^ 0. Since e^ 
in not injective, Vq fl y is nonempty, and contains a rank-1 element Mi. Let vi E V be 
such that Mi(vi)(vi) 7^ 0. By Lemma 4.3, we can then find a rank-1 element element 
M 2 G uj 1 -. 

Clearly Mi and M 2 are independent. Let Z be the 2-plane they span, and let 
W = ker(Mi) n ker(M 2 ). W C V is codimension two and Z C W" 1 . Let i7 be 
another element of Y, independent of Mi and M 2 , so that Y is the span of Mi, M 2 , 
and H. 

By Proposition 4.4, there exists t E V such that 

(18) et:Z^V* 

is injective. Injectivity is an open condition, so et is injective on Z for general t. However, 
by assumption et is never injective on Y . Thus for each t there exists M3 G Z such that 
M 3 (t) = #(£). However, that implies that, for every w E W, 

(19) H(w)(t) = H(t)(w) = M 3 (t)(w) = M 3 (w)(t) = 0. 

Since this is true for general t, H(w) must be zero, so H E W^. But then Y C W 1 - . 
Finally, the dimension of W 1 - is exactly 3, so X = Y = W^. □ 

For i > 3, we will prove Theorem 2.5 by induction on i. The key to the inductive 
step is the following lemma: 
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4.7 Lemma. Suppose that Theorem 2.5 applies for all values of i < k, and that X is 
a subspace of Sym(V, V*) satisfying the hypotheses of theorem (1-4) with i = k + 1. Let 
M G X,v G V be such that M(v)(v) ^ 0. There is a g — i + 2- dimensional subspace 
W C V , containing v, such that 

(20) w-*- = v ± nx <zS(v,v*) 

In particular v - 1 fl X contains non-zero elements of rank one. 

Proof. If W C V is a linear subspace, and W 1 - C S(V, V*) is the subspace of maps / 
with f\ w = 0, then the natural map W 1 - — > Hom(V/W, (V/W)*) identifies W 1 - with 
S(V/W, (V/W)*). 

Let X v : = D 1 nl. By assumption, e v is not injective on every z-dimensional subspace 
Y G X. This implies that the rank of e v : X — > V* is at most % — 1, so the kernel of e„ 
has co dimension at most % — 1. Thus 

dim(X„) = dim(X) - codim(X„, X) > ^ ~ ^ - (i - 1) 

(2D = 

We claim that X„ C S(V/v , (V/v)*) satisfies the evaluation condition of Theorem 
2.5. If not, then for some ro/0, the range of e w on X v has dimension i — 1, and so 
equals the range of e w on all of Y - . Note that e w = e w+v on X v C v 1 , so (after possibly 
replacing w by to + v ) we can assume M(ty)(v) 7^ 0. But now M(w) = N(w) for some 
iV G w" 1 ", thus M(w)(v) = N(w)(v) — N(v)(w) = 0, a contradiction. 

We now apply Theorem 2.5 to X v . Viewed as an element of S(V/v, (V/v)*), X v is 
the orthogonal complement of a (g — i + 1) -dimensional subspace Wq C V/v. This is 
tantamount to equation (20), where W is the preimage of Wo- □ 

Proof of Theorem 2.5. In light of Propositions 4.4, 4.5 and 4.6, we need only prove 
Theorem 2.5 for i > 3 and for X of dimension exactly i(i — l)/2. We do this by 
induction on i, relying on the previously proven base case i = 3. 

To establish the induction, we apply Lemma 4.7 to get a rank one element M G X. 
Choose v £ V with M(v)(v) 7^ 0, and apply Lemma 4.7 again, so that 

(22) W ± = X v cX 

for some g — i + 2-plane W C V that contains v. Now let W = ker(M) fl W. We claim 
that X = W ± . 
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To see this, let Z C X be the span of W and M, and let t be an element of V that 
is neither in W nor in the kernel of M. (This is an open condition). Note that M(t) is 
a nonzero multiple of M(v), so M (£)(?;) 7^ 0. 

Since t g W, e t : W 1 - -> has rank exactly % - 2. Also M(t) E~ e t (W L ), since that 
would imply M(t)(v) = 0. Thus e t (Z) C has dimension z — 1 and so is equal to 
e t (X). Now let a; be an arbitrary element of X. Since e t (X) = e t (Z), there exists an 
element z E Z C VF" 1 such that x(t) = z{t). But then, for any w E W, 

(23) x(w)(t) = x(t)(w) = z(t)(w) = z(w)(t) = 0. 

Since t was chosen arbitrarily from an open set, x(w) must be zero. Since x and w were 
arbitrary, X C W^. But X and W 1 - have the same dimension, so X = W^. □ 

§5 The tangent space is constant. 

5.1 Theorem. Let X C 7i g 6e a variety of dimension ^ , with i > 3, such that at 
every smooth point A e X there is a (g — i + 1) -plane W(A) C C 9 such that 

(24) T A X = W(A) 1 - c T A H g = S g . 
Then W(A) = W is constant and X C H g is the affine subspace 

(25) X = {M e H g \M\ w = t\ w } 
for some fixed r G Ti g . 

Proof. Once VF(A) is constant, the tangent space is constant and equation (25) follows. 
Let I := {1, . . . , i — 1}, K = {i, . . . ,g}. Pick a smooth point Aq. After changing basis we 
can assume VF(A ) is the span of {efc}, k E K, where ei, . . . , e g E C 9 are the standard 
basis elements. Then on an open set U around A the coordinates A a b of the upper 
left (i — 1) x (i — 1) block are analytic coordinates for X, and there exist holomorphic 
functions z a ,t 7 a E I,t E K so that 

(26) v t (A) :=e t + J2 z s,te s 

sEl 

form a basis of W(A). We will show that d a bZ St t = for all a,b,s E I,t E K (where 
dab means differentiation with respect to the coordinate A a ^)- At A E U a basis for 
TaX C S g is given by the matricies d a bA. Note that the upper-left-hand entries of d a bA 
are particularly simple. If a, b,c,d E /, then 

(27) (d a bA)cd = 1 if (a = c and b = d) or (a = d and b = c), otherwise. 
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By assumption (d ab A) ■ v t = 0. Looking at the first i entries of this product gives: 

d ab A C!t = a,6,ce /,£ e K,cg {a,b} 

(28) z a , t + da b A btt = a,beI,teK 

If i > 3, or if i = 3 and a = b, then there exists c e I \ {a, 6}. Thus for s 6 /, t G -fC, 

(29) <9 a 6^,t = -<9 a6 (d sc A C)t ) = -d sc (d ab A c , t ) = 0. 
Finally, if % = 3 and a, 6 G / we have 

d a bZ a ,t = —dab(d aa A aj t) 
= -d aa (d ba A a! t) 

(30) = a aa z M = on 

§6 Compactness. Here we argue that the image of X(W, r) in A g cannot be compact. 
The meaning of X(W, r) is clearer in an alternative realization of 7i g : 

Tig, with its Sp(2<7, M) action is canonically identified with the set of complex struc- 
tures on M? 9 = C 9 compatible with the standard symplectic form (,), where Sp(2<7, M.) 
acts on complex structures by conjugation. For M e Tig the corresponding complex 
structure on R 2ff is given by identifying M? 9 with C 9 by sending a column vector igC 9 
to 

(31) f M (x) = (Re(x),-Re{Mx)). 

The above, and the next Lemma, are all immediate from [Mumford83,4.1] 
6.1 Definition-Lemma. For M e X(W,t) 

(32) V := f M (W) = f T (W) 

is a non- degenerate subspace of M. 2g with a fixed complex structure compatible with the 
restriction of (,). Under the above realization of Tig, X(W, r) is identified with the 
set of complex structures on M? 9 , compatible with (,), extending the complex struc- 
ture structure on V . This is naturally identified with compatible complex structures on 
(y- 1 , and thus with H r , where r is the complex codimension ofW. 

Let H (resp V C H) he the subgroup of Sp(2<7, R) (resp Sp(2g, Z)) that preserves 

X(W,T) - or equivalently by the above, the subgroups that preserve, and act complex 

linearly on V. Restriction to V, V 1 - identifies H with U(s) x Sp(2^,R), and under this 

identification, U(s) x {1} is the stabilizer of r. Recall that a subgroup of a semi-simple 

Lie group is called a lattice if it is discrete, and the quotient has finite volume. It is 

called co compact iff the quotient is compact. The following is clear: 
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6.2 Lemma. Assume the image, Z, of X(W,r) in A g is a closed analytic subvariety. 
Then 

(33) Y\(H/V(s))=Y\X(W,r) 

is the normalization of Z , Y C H is a lattice, and cocompact iff Z is compact. 

As U(s) is compact, r C H is a lattice iff its image in H/XJ(s) = Sp(2r, R) is a 
lattice. Furthermore, by the Borel Density Theorem, [Zimmer84,3.2.5], a lattice in a 
semi-simple Lie group without compact factors is Zariski dense. 

Thus it sufficies to prove the following, which was done jointly with (and mostly by) 
Scot Adams: 

6.3 Theorem. Let W C R 2ff be a real subspace, of codimension 2r, together with a 
complex structure compatible with (, )\w- Let H C Sp(2<7, R) be the subgroup of elements 
that preserves, and acts complex linearly on, W . Let Y := H n Sp(2g, Z). The image of 
the restriction map 

(34) T -> Sp(W ± , R) = Sp(2r, R) 

is Zariski dense iff W is spanned by integer vectors. In this case the image is a non- 
cocompact lattice. 

Proof. Suppose first W has a basis of integer vectors. Then the image of Y in H/ U(s) = 
Sp(2r, R) is commensurable with (i.e. up to finite index subgroups equal to) Sp(2r, Z), 
a non-cocompact lattice. 

Now suppose that W is not spanned by integer vectors, and that the image of Y is 
dense. Let G C H C Sp(2g, R) be the Zariski closure of Y. Restriction to W 1 - gives a 
surjection 

(35) G(R) -> Sp(W ± , R) = Sp(2r, R) 
which together with restriction to W defines an embedding 

(36) G(R) C U(s) x Sp(2r, R). 

For any complex subspace V C C 2fl , non-degenerate with respect to the complex 

linear extension of (, ), let Gy C Sp(2g, C) be the subgroup of matricies that act trivially 
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on V. Of course restriction to V 1 - identifies this with Sp(F ± ,C). We indicate its 
(complex) Lie algebra by 

(37) S p F iCsp(2 9 ,C). 

We abuse notation and refer to W ®m C as W as well. 

Note that if a Lie subalgebra t) C sp(2<7, C) is stable under Gal(C/L) for a subfield 
L C C, then [) is defined over L, i.e. is the extension of scalars of a Lie subalgebra of 
sp(2<7,L), which we denote l)(L). The subalgebra t)(L) C h is simply the subset fixed 
by Gal(C/L). 

Since W is not spanned by rational vectors, there exists an element a G Gal(C/Q) for 
which a(W) ^ W. Let V = a(W). Let g C sp(2g, C) be the complexified Lie algebra 
of G. 

By Levi's theorem, [Serre65,4.1], (35) has a section at the level of real Lie algebras. 
Since there is no non-trivial homomorphism sp(2r, M.) — > u(s) we conclude from (36) 
that sp w ± C g. Note the image is an ideal, as it corresponds to the kernel of the 
restriction 



(38) G — > U(s) C Sp(W, M). 
If V = V, let 

(39) r = sp v ± + sp w ± C sp(2g, C). 
Since Gal(C/Q) preserves T, it preserves g, and thus 

(40) sp v ± = a(sp w ±) C g 

is another ideal. Since sp v ± and sp^/^ are distinct simple ideals, their intersection is 
trivial, so r is the direct sum 

(41) r = sp v ± © sp w ± C g, 

and is itself an ideal. 
IfV^V, let 



(42) r = sp v ± + sp v ± + sp w ± = sp v ± +sp v ± + sp w ± C sp(2£r, C). 
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By the same reasoning as before, each factor is a simple ideal, so r is the direct sum of 
its factors, and is an ideal. 

In either case, we have an ideal r C g. Since r, g are preserved by complex conjugation, 
we have an induced real ideal r(R) C g(R). By construction r(R) is isomorphic to 

sp(2r, R) © sp(2r, R)ifV = V 

(43) sp(2r, C) © sp(2r, R)ifV^V 

By (35) there is an induced surjection of real lie algebras 

(44) r(R) -> sp(2r, R) 

whose kernel, by (36), is a subalgebra of u(s). As r(R) is semi-simple, the kernel is a 
direct sum of simple factors of r(R). Since no factor has a non-trivial homomorphism 
to u(s), we conclude r has only a single factor, which is a contradiction. □ 

§7 Remarks on characteristic p > 0. 

In positive characteristic the only known codim g complete subvariety of A g is the 
locus Z C A g © F p of Abelian varieties of p-rank zero, discovered by Oort. Results of 
[Koblitz75] imply that the tangent space to Z is just as in Corollary 2.6 (with % = g): 

7.1 Proposition (Koblitz). At points [A] in a Zariski dense subset of Z the following 
hold: The p-linear Frobenius 

(45) H:H 1 (A,O a )^H 1 (A,O a ) 

has one dimensional cokernel. Let L^j C H°(A,Q A ) = E^j be the dual line. Z is 
(orbifold) smooth at [A] and its tangent space is given by 

(46) T [A] Z = Lf A] c S(E [A] , E* [A] ) = T [A] A g . 

Proof. This follows from [Koblitz75]. Here we give a few details for the readers conve- 
nience. The p-linear Frobenius gives a commutative diagram 



► H°(A,n A ) ► H$> R (A) ► H\A,O a ) > 



(47) 



H 



H°(A, fl A ) ► H^ R (A) > H\A.O A ) > 
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The snake lemma induces a canonical map B : ker(H) — > H°(A,Q\). This is the 
(restriction to ker(H) of the) map given by the matrix B in [Koblitz75]. The argument 
on pages 188-189 shows that the tangent space to Z is the perpendicular to the (one 
dimensional) image of B. So it remains to show that the image of B is dual to the 
cokernel of H. This follows from the fact that the image of F is isotropic for the 
canonical pairing on H^ R . □ 

Given the parallel between Theorem 7.1 and our argument in characteristic zero, it 
is natural to wonder: 

7.2 Question. If Z C A g <S> F p is complete and of codimension g is the tangent space 
to Z as in Corollary 2.6? Is Oort's example the only possibility? 

§8 M c g . 

Proof of 1.2.1. By [Diaz87] a compact subvariety of M g or M g 1 has codimension at 
least g, for any g, and a compact subvariety of M g or M g ^ has codimension at least 
2g — 1. Now assume g > 3 and let Z C M g be a compact subvariety of codimension at 
most g. We have a regular map 

M c g - A ff 

with zero dimensional fibres outside of <9M^ (meaning the complement of M g C Mp. 
Suppose first (7 = 3. By (1.2) Z C <9M| and so projects to a complete surface in M|, 
violating Diaz's bound. So we may assume g > 4. We proceed by induction. 

By Diaz's bounds Z must meet the boundary. Let Zi be the intersection of Z with 

If Zi is non-empty we have by Diaz's bounds 

g > codim(Z, M g ) > coding, Si) 
> codim(7r i (Z i ),M? 1 ) + codim(7r fl _ i (Z i ), M^^) 

where Hk indicates the projection onto the factor Mj: 1 . Furthermore, by induction, we 
can replace the last term by g + 1 if either i or g — i is at least 3. Thus the only possibility 
is that g — 4 and Z meets only 82 . But then its image in M| 1 is a complete surface 
contained in il^i, contradicting Diaz's bounds. □ 
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